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1 . INTRODUCTION 


This  report  describes  a comparative  stud^by  ARINC  Research 
Corporation '-of""six  alterriati've~facllitie's 'Tor  system- level  eval- 
uation at  the  Navy  Electronics  Laboratory.  This  study  differs 
-in  two  ways  from  past  analyses  of  alternative  facilities.  First, 
the  intended  uses  of  the  facility  are  redefined  to  include  the 
evaluation  of  shipboard  electronic  systems  other  than  the  com- 
munications system  for  which  the  facility  was  originally  planned. 
Second,  the  functional  capabilities  of  each  alternative  facility 
are  compared  from  a cost-effectiveness  standpoint . ^ 

Admittedly,  the  above-mentioned  *^relatipnship^ necessitates  a 
certain  degree  of  subject lvenes;^/^T^)r  this  reason,'^an  attempt  is 
made  to  provide '^^-d4sp-l-ay>the  information  in  such  a way  as  to 
allow  the  reader  to  introduce  his  own  subjective  estimates  and 
reformulate  the  results.  Any  effectiveness  model  will,  in  a like 
manner,  have  some  subjective  characteristics  which  are  introduced 
to  bridge  gaps  in  Information.  In  this  study,  Bayesian  weighting 
factors  were  used  as  a means  of  introducing  experience  factors 
where  the  accumulation  of  factual  data  was  Impractical.  However, 
in  the  application  of  these  weighting  factors,  the  effect  of  each 
on  the  modified  cost  of  ownership  of  the  systems  to  be  evaluated 
was  taken  into  consideration. Thus  the  effectiveness  of  the 
alternative  facilities  is  related  to  the  impact  of  system  evalu- 
ation on  the  cost  of  ownership  tod  delay  in  delivery  of  each 
system. 

The  intent  of  this  study  is  to  present  sufficient  data  to 
point  up  the  differences  between  the  alternative  facilities. 

Based  on  the  significant  differences,  the  most  desirable  alterna- 
tive can  be  selected. 
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2. 


SUMMARY 


2. 1  Requirements 

Certain  requirements  were  chosen  as  criteria  for  the 
evaluation  of  the  six  alternative  facilities.  These  requirements 
are  discussed  in  sections  2.1.1  through  2.1.3. 

2.1.1  Technical  Requirements 

Table  2-1  summarizes  the  technical  requirements  for  the  test 
facility.  Each  requirement  is  expressed  two  ways:  as  minimum 
acceptable  and  as  desirable  but  not  mandatory.  A facility  must 
possess  at  least  the  minimum  characteristics  to  be  acceptable  as 
an  alternative.  Each  acceptable  alternative  is  then  graded  as 
to  its  effectiveness  in  providing  the  desirable  characteristics. 
The  method  of  grading  is  described  in  section  4;  the  rationale 
supporting  the  method  is  discussed  in  section  3. 

2.1.2  Space  Requirements 

The  minimum  floor  space  required  for  the  facility  was 
estimated  as  follows: 


SACOM  NSACS/NFF 

10,000  ft 2 

Additional  and  supporting 
systems 

20,000 

Administrative  (security  office, 
guard,  etc.) 

1,000 

Personnel  comfort  (rest  areas, 
restrooms) 

1,000 

Air  conditioning,  switchboard, 
power  station,  etc. 

500 

+10  percent  for  passageways, 
entrances,  etc. 

3,250 

Total: 

35.750  ft^ 

To  allow  for  growth,  a capacity  for  expansion  to  twice  this 
size  is  considered  desirable. 

2.1.3  Alternative  Facilities 

The  alternative  facilities  are  described  in  section  2.2. 
Note  that  although  Battery  Ashburn  is  categorized  as  an  existing 
building,  a new  building  of  26,300  square  feet  will  be  required 
to  meet  the  minimum  space  requirements. 
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TABLE  2-1 

SUMMARY  OF  TEST  FACILITY  REQUIREMENTS 


Requirement 

Minimum 

Acceptable  Characteristics 

Desirable 

Characteristics 

Floor  Space 

35,750  ft^ 

Expandable  to  71,500  ft^ 

Accessibility 

Location  convenience 

Reachable  by  private 
transportation 

Reachable  by  both  public 
and  private  transportation 

Proximity  to  NEL 

Not  greater  than  30  minutes 
traveling  time  (by  auto- 
mobile) from  NEL 

Less  than  15  minutes  trav- 
eling time  (by  automobile) 
from  NEL 

Parking 

Space  for  100  cars 

Expandable  to  accommodate 

300  cars 

Entryway 

Work  spaces  can  receive 
6'x4'x4'  objects  without 
modification 

Work  spaces  can  receive 
6'x8'xo'  objects  without 
modification 

Elevators 

2-ton  capacity,  40  ft®  area; 
running  to  ali  floors  and 
roof 

4-ton  capacity,  6o  ft®  area; 
running  to  all  floors  and 
roof 

Helipad 

Helicopter  can  land  by 
prearrangement 

Helicopter  can  land  without 
prearrangement 

Microwave  line 
to  NEL 

Possible  with  one  repeater 
on  existing  Government 
site 

Direct  line  of  sight  to  NEL 

Security 

Secret 

Secret  area  does  not 
require  additional  guard 

Secret  area  does  not 
require  additional  guard 

Top  Secret  (T/S) 

Top  secret  area  feasible 
with  one  additional  guard 

Top  secret  area  feasible 
without  extra  guard 

Antenna  Facility 

Scope 

Room  for  approximately  one 
full-scale  NFF  suit 

100  percent  additional  area 
for  evaluation  and 
experimentation 

Counterpoise 

Copper  mesh 

Simulated  ship's  deck 

View 

No  external  obstruction 
above  10°  elevation 

No  external  obstruction 
above  5“  elevation 

Proximity 

No  objects  above  roofline 
within  50  feet 

No  objects  above  roofline 
within  500  feet 

RF  Noise  Environment 

Noise  Level 

(Per  specification) 

(Per  specification) 

Restriction 

Can  radiate  10  kw  below 

30  Me,  1 kw  above  30  Me 

No  limit 

Packaging  Flexibility 

Sufficient  to  allow  relo- 
cation of  screened  areas 

Can  be  readily  reparti- 
tioned to  accommodate  full- 
scale  mock-up 

Aesthetic  Qualities 

Not  less  than  minimum 
existing  in  same  location 

Better  than  adjacent 
facilities 
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2.2  Facility  Alternatives 


2.2.1  Possibilities 

In  selecting  the  test-facility  alternatives,  an  attempt  was 
made  to  narrow  the  field  as  much  as  possible.  An  extreme  case  of 
only  two  alternatives  ("in  the  Bunker  Hill" . "in  a building")  was 
examined  and  found  to  be  the  simplest  to  analyze  but  the  least 
Informative.  The  other  extreme  was  to  weigh  all  possible  choices. 
Including  all  those  previously  considered  (see  Table  2-2).  Although 
this  approach  would  be  the  most  Informative,  time  limitations  and 
the  complexity  of  data  gathering  and  analysis  indicated  that  a 
moderate  number  of  most  likei.y  choices  should  be  considered.  Even 
then,  each  choice  had  to  be  carefully  defined  to  prevent  escala- 
tion into  a large  family  of  subalternatives . 

The  approach  used  throughout  this  study  has  been  to  ensure 
flexibility  sufficient  to  allow  other  alternatives  to  be  con- 
sidered in  the  decision  matrix. 


2.2.2  Alternatives  Selected 


Three  major  alternatives,  each  divided  into  two  subalterna- 
tives, were  designated  as  follows: 

(1)  Bunker  Hill 

A.  At  Pier  F,  NAS,  North  Island 

B.  Adjacent  to  the  east  shore  of  Pt.  Loma 

(2)  New  Building 

A.  West  side  of  Pt.  Loma,  water-level  building 

B.  Building  within  the  present  NEL  compound 

(3)  Existing  Building 

A.  Battery  Ashburn  (with  additional  construction) 

B.  Leased  building 

2.2.3  Description  of  Alternatives 

lA  — The  Bunker  Hill  is  an  aircraft  carrier  removed  from  the 
reserve  fleet  and  made  available  to  the  NEL  as  a possible  system 
facility.  It  has  been  partially  prepared  for  such  use,  and 
electromagnetic  research  studies  are  presently  being  conducted 
onboard.  In  its  present  location  (Pier  F,  North  Island)  it  will 
be  considered  as  an  alternative. 
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Facility  Afloat  Facility  Ashore  Geographical  Location 


New  Building 

Existing  Building 

Ft . Loma 

NEL  Compound* 

Waterfront,  west  side* 

Naval  repair  facility 

Border  field  station 

Battery  Ashburn 

Atop  Ft.  Loma,  2 miles 
west  of  NEL  compound* 

Leased  Building 

Within  15-mile  radius  of 
NEL  compound* 

Barge 

Moored  west  of  Ft . Loma 

Bunker  Hill 

Breakwater  west  of 

Ft . Loma 

With  causeway 

Without  causeway 

Fier  F,  North  Island* 

East  side  of  Ft.  Loma, 
south  of  Ballast  Foint* 

Moored  west  of  Ft.  Loma, 
without  breakwater 

Operating  Ship 

Fier,  Ft.  Loma 

(NEL  control) 

Without  pier,  San  Diego 
harbor 

Operating  Ship 
(Line) 

On  request 

Towert  (2) 

West  side  of  Ft.  Loma 

*Consldered  in  ARINC  Research  study. 

tSame  as  afloat  from  electromagnetic 

standpoint. 
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IB  — There  is  a possibility  that  the  Bunker  Hill  can  be 
moved  to  the  sheltered  east  shore  of  Pt.  Loma.  A preliminary 
examination  of  the  available  space  indicates  that  the  partially 
sheltered  coastline,  south  of  Ballast  Point  and  close  to  the 
cliffs,  is  the  only  space  still  available  for  mooring  a ship  as 
large  as  the  Bunker  Hill. 

2A  — A water-level  building  on  the  west  shore  of  Pt.  Loma 
has  been  considered  in  previous  studies.  This  location  would 
partially  simulate  the  electromagnetic  and  RF  radiation  charac- 
teristics of  an  actual  ship.  A lead-coated  roof  and  a tower  are 
included  in  the  specifications  to  simulate  a suitable  ground 
plane.  Rough  estimates  were  used  in  arriving  at  the  cost  of  this 
building. 

2B  — A new  building  located  atop  Pt.  Loma,  within  the  NEL 
compound,  has  been  specified  and  requested  as  MILCON  No.  P027- 
Although  this  building  was  not  specifically  intended  as  an  NEL 
system  test  facility,  all  or  part  of  it  could  be  so  designated 
with  little  change  in  plans. 

3A  — Battery  Ashburn  is  an  underground  complex  located  on 
Pt . Loma  within  easy  commuting  distance  (5  minutes)  of  the  cen- 
tral compound.  Although  most  of  the  space  has  been  committed 
to  other  projects,  the  Battery  could  be  made  available,  either 
in  part  or  in  total,  for  the  facility.  Considering  that 
9,000  square  feet  of  existing  space  is  available,  a new  building 
of  26,750  square  feet  would  be  required  atop  Battery  Ashburn. 

3B  — Many  buildings  are  available  in  San  Diego  on  a lease 
basis.  This  possibility  should  be  considered  in  that  it  repre- 
sents a considerable  variance  from  previous  considerations.  A 
cost  effectiveness  study  may  reveal  that  a leased  facility  is 
the  most  economical  and  effective. 

2.3  Summary  and  Conclusions 

The  results  of  this  study  indicate  quite  clearly  that  any 
conclusions  reached  must  be  conditional — i.e.,  decisions  are 
dependent  upon  a number  of  differing  premises  available  to  the 
decision-maker.  Table  2-3  ranks  the  alternatives  in  terms  of 
cost,  effectiveness,  and  irreducible  factors.*  It  should  be 
noted  that  the  irreducible-factor  rankings  could  vary  from  those 
in  Table  2-3>  depending  upon  the  relative  weights  assigned  to 
each  factor  by  the  decision-maker. 


*Irreducible  factors  are  those  which  cannot  be  directly  reduced 
to  monetary  terms;  see  section  2.3.3* 
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TABLE  2-3 

RELATIVE  RANKING  OF  ALTERNATIVE  FACILITIES* 
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2.3.1  Cost  Ranking 


Table  2-3  gives  two  cost  rankings.  Under  "Minimum  Space"  the 
ranking  is  premised  on  an  assumption  that  the  minimum  acceptable 
floor  space,  35>750  square  feet,  will  remain  the  same  over  the 
ten-year  base  of  this  study.  From  this  standpoint.  Battery 
Ashburn  (Alternative  3A)  including  the  required  new  construction 
is  the  least  costly,  followed  in  order  by  the  MILCON  building  (2B) 
and  the  Bunker  Hill  at  Pier  P (LA). 

In  the  second  column  the  rankings  are  premised  upon  an  assump- 
tion of  a doubled  floor-space  requirement  within  five  years.  On 
this  basis.  Bunker  Hill  at  either  location  and  Battery  Ashburn 
become  nearly  equal  in  cost,  with  the  latter  slightly  lower. 

However,  these  rankings  could  well  lose  much  of  their  signifi- 
cance when  considered  in  light  of  the  suggested  financial- support 
approach  discussed  in  section  7-  With  this  approach,  ajid  assuming 
that  a 90  percent  utilization  of  the  facility  can  be  achieved, 
annual  support  costs  at  NEL  would  be  in  the  $15,000-$30,000  range, 
which  considerably  reduces  the  dollar  differences  among  the  alter- 
native facilities.  The  dominating  monetary  factor  would  then  be 
the  acquisition  cost  (see  Table  7-1,  column  1).  When  viewed  in 
this  manner,  the  Bunker  Hill  at  Pier  F (first)  and  the  leased 
building  (second)  far  outrank  the  other  alternatives. 

2.3.2  Effectiveness  Rankings 

The  analysis  of  section  4 indicates  that  the  Bunker  Hill  at 
either  location  (Alternative  LA  and  IB)  would  be  an  estimated 
24  percent  more  effective  than  either  of  the  remaining  four  alter- 
natives in  the  identification  and  resolution  of  system  uncertainties. 

In  view  of  the  results  summarized  above,  one  question  remains 
unanswered:  How  Important  is  the  24  percent  greater  effectiveness 
of  the  Bunker  Hill  as  a system  facility  as  measured  against  the  cost 
differential  of  the  carrier  and  the  lowest  overall  priced  alterna- 
tive? The  results  of  section  5 can  be  reexamined  to  shed  light 
upon  this  question.  It  was  shown  that  for  a system  similar  to 
those  being  considered  for  system- level  evaluation,  the  quality  of 
the  facility  made  a difference  of  approximately  $7  million  in  the 
expected  risk  cost  ($16.12  million  for  a marginal  facility  minus 
$8.58  million  for  an  adequate  facility).  It  is  not  suggested  that 
the  building  and  the  hull  alternates  correspond  to  the  "marginal" 
and  the  "inadequate"  facilities  of  Table  5-1.  However,  the  expected 
risk  of  $8.58  million  corresponding  to  the  "adequate"  facility  in 
that  figure  might  be  used  to  adjust  the  modified  risk  totals  of 
Table  4-1  to  represent  reasonable  absolute  as  well  as  relative 
values.  Thus,  if  the  modified  risk  of  $64.6  million  for  the 
Bunker  Hill  is  adjusted  downward  to  equal  the  $8.58  million  of 
Table  5-lj  then  the  $80.4  million  risk  figure  corresponding  to  the 
building  alternates  becomes 

^|^^($8.58  million)  = $10.6  million. 
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The  expected  savings  offered  by  the  Bunker  Hill  for  each  system 
would  then  be  $10.6  million  minus  $8.58  million,  or  $2.0  million. 
For  five  systems  evaluated  within  the  facility,  the  expected 
savings  would  be  five  times  this  value,  or  $10  million. 

The  differential  cost  of  retaining  the  Bunker  Hill  rather 
than  the  lowest  total-cost  building  alternative  (Battery  Ashburn) 
is  $493»800  (see  Table  2-3)  for  a 10-year  period.  The  trade-off 
is  now  evident  between  investment  and  return.  The  results  indicate 
that,  on  the  basis  of  total  costs  only,  an  additional  facility 
investment  of  $493j800  by  the  Navy  can  be  expected  to  yield  a 
$10  million  return  in  benefits  to  participating  systems.  This 
conclusion  is,  of  course,  valid  only  within  the  assumptions  of 
sections  4 and  3)  and  is  based  upon  the  concept  of  expectation 
according  to  probability  theory. 

2.3.3  Irreducible  Factors  (See  §3*3.4) 

The  most  significant  of  the  Irreducible  factors  appears  to 
be  the  immediate  availability  of  the  Bunker  Hill.  This  availability 
factor  is  critical  relative  to  the  planning  of  system  evaluations 
for  projects  in  the  development  phase  at  the  present  time.  Par- 
ticularly in  case  of  the  integrated  communication  system,  equip- 
ment is  already  being  delivered  to  NEL  for  preliminary  system 
evaluation  purposes.  All  building  alternatives  will  require  a 
minimum  of  three  years'  lead  time  before  occupancy.  Thus  the 
choice  of  a building  alternate  would  mean  a period  of  no  facility 
prior  to  construction,  or  a transition  period  utilizing  the  Bunker 
Hill  until  completion  of  construction.  The  move  from  ship  to 
building  would  create  an  awkward  interruption  of  evaluation 
activity.  In  the  case  of  Battery  Ashburn  (3A),  the  interval  of 
interruption  could  be  reduced  if  the  existing  7,000  square  feet 
of  space  were  used  to  extend  the  move  from  ship  to  building  over 
a longer  period-- say  a year  or  more. 

Other  significant  factors  accounting  for  the  rankings  in 
Table  2-3  are  expansion  flexibility,  the  degree  of  approximation 
of  the  real  environment  in  which  the  systems  under  evaluation 
will  eventually  be  operated,  and  the  cost-reduction  potential 
which  obtains  to  Bunker  Hill  through  utilization  of  existing 
resources . 

All  of  these  factors  combine  to  give  Alternatives  lA  and  IB 
a substantial  lead  in  terms  of  irreducible  factors. 

2.3.4  Support  of  Basic  Assumption 

The  basic  assumption  of  this  study  is  that  a system  evalua- 
tion facility  is  needed  at  NEL.  An  example  in  section  5 Indicates 
the  tremendous  cost-  and  time-saving  potential  for  large  systems 
through  Investment  in  a system-level  evaluation  prior  to  operation 
evaluation  (OPEVAL).  It  is  axiomatic  that  such  an  evaluation 
requires  a facility  in  which  it  can  be  performed.  Section  5 
therefore  provides  strong  validation  of  the  basic  assumption. 
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2.3.5  Conclusions 


From  a technical  standpoint,  the  Bunker  Hill  is  an  estimated 
24  percent  more  effective  than  either  new-building  alternate.  The 
cost  of  all  six  alternatives  over  a 10-year  period  is  within  l4  per- 
cent of  their  mean  value,  with  Battery  Ashburn  (including  new  con- 
struction) being  the  lowest  in  cost.  In  the  case  of  expansion  after 
five  years  to  twice  the  minimum  required  floor  space,  the  cost  of 
the  Bunker  Hill  approaches  equality  with  that  of  the  Battery  Ashburn 
because  of  the  ship's  surplus  of  unused  space.  Furthermore,  the 
Bunker  Hill  is  Immediately  available,  while  the  building  alterna- 
tives pose  a 2-  to  3-year  delay  period  for  funding  and  construction. 

The  results  summarized  in  the  previous  paragraphs  appear  to 
indicate  strongly  the  desirability,  all  things  considered,  of 
retaining  the  Bunker  Hill  and  continuing  its  preparation  as  a 
facility  for  evaluating  all  complex  Navy  shipboard  electronic 
systems  both  individually  and,  where  practical,  collectively. 

A study  now  in  progress  by  the  Bureau  of  Yards  and  Docks  will 
determine  whether  or  not  the  more  desirable  location  south  of 
Ballast  Point  can  be  prepared  for  the  Bunker  Hill  within  an 
acceptable  cost. 
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3. 


TECHNICAL  APPROACH 


This  section  describes  the  approach  taken  for  comparing  the  six 
alternative  system-evaluation  facilities  described  In  section  2.2. 

The  cost  effectiveness  of  a test  facility  Is  evaluated  In  terms 

of; 

(1)  Its  direct  cost  In  dollars;  and 

(2)  Its  effectiveness  with  respect  to  the  degree  of  Impact  on 
the  overall  schedule  and  cost  of  ownership  of  the  systems 
served. 

In  this  study  the  cost  of  the  facility  was  modified  to  Include 
only  those  expenditures  considered  significant  In  evaluating  dif- 
ferences In  alternatives. 

Section  3«1  defines  the  relationships  Involved  In  the  applica- 
tion of  system-level  evaluation  to  development  projects. 

Section  3.2  defines  the  measures  of  effectiveness  of  a system  evalu- 
ation facility.  Section  3*3  compares  the  cost  of  alternative 
facilities . 

3 . 1 System  Development  and  the  System 
Evaluation  PhasT 

The  primary  purpose  of  a system  evaluation  facility  Is  to 
optimize  system  design  parameters  In  a controlled  environment  as 
representative  of  an  operational  environment  as  Is  practical.  In 
some  facets  of  system  operation,  optimization  Involves  simply  the 
adjustment  of  parameters.  In  other  areas.  It  Involves  the  orderly 
Identification  and  resolution  of  technological  and  requirements 
iincertalntles  which  theretofore  were  unidentified  or  unresolved. 

The  latter  process  Is  the  key  to  the  effectiveness  of  a system  evalu- 
ation facility.  Just  how  this  relationship  Is  obtained  Is  discussed 
In  the  paragraphs  which  follow. 

The  R&D  phase  of  system  development  generally  Includes  mile- 
stones for  Identifying  and  resolving  problems  and  \ancertalntles . 
These  milestones.  Illustrated  In  Figure  3-lj  represent  successively 
more  refined  estimates  of  the  system's  Intrinsic  effectiveness,  Ej. 
As  shown  In  the  figure,  the  range  or  spread  of  the  estimated  effec- 
tiveness, F,  reflects  the  confidence  In  the  estimate.  Another 
quantity,  Emin,  represents  the  minimum  effectiveness  considered  to 
be  acceptable  for  the  mission  requirements. 

Each  successive  milestone  In  the  R&D  phase  depicted  In 
Figure  3-1  Is  aimed  at  narrowing  the  confidence  limits  of  the  effec- 
tiveness estimate.  At  the  first  point  In  time  where  the  confidence 
limits  do  not  Include  the  required  minimum  effectiveness  point,  a 
significant  cheuige  In  project  direction  must  be  accomplished 
(Project  "b".  Figure  3-1).  Either  the  system  concept  must  be 
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Confidence  Limits  for  E 


Defined  Milestone  Milestone 


Eval. 
Milestone 


Milestone 


SYSTEM  DEVELOPMENT  PHASE  (TIME) 


E^  = Inherent  effectiveness  of  system 

^mlh  ” Minimum  acceptable  system  effectiveness 

E = Predicted  or  measured  system  effectiveness 
at  time  t in  R&D  phase 


FIGURE  3-1 

MILESTONES  IN  SYSTEM  DEVELOPMENT 
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changed  to  improve  its  capabilities  or  the  minimvun  effectiveness 
requirements  must  be  reassessed,  with  the  possibility  that  a lower 
minimum  requirement  may  be  acceptable. 


System-effectiveness  confidence  limits  are  narrowed  as  uncer- 
tainties are  resolved.  In  general,  the  earlier  each  mcertalnty  is 
resolved,  the  lower  is  the  overall  program  cost.  The  same  is  true 
for  the  time  of  delivery*  of  an  acceptable  system.  Dollar  savings 
thus  achieved  can  be  thought  of  by  the  procuring  agency  as  a 
decrease  in  the  cost  of  system  ownership.  Unresolved  uncertainties 
pose  a risk  to  the  project  in  terms  of  increases  in  the  overall  cost 
of  ownership  of  a system.  This  risk  function  exhibits  a distinct 
discontinuity  during  operational  evaluation  (OPEVAL),  as  reflected 
in  Figure  3-2.  Prior  to  OPEVAL,  the  consequences  of  design  uncer- 
tainties can  be  confined  to  design  changes  within  an  imreleased 
specification  package.  During  OPEVAL,  the  resolution  of  imcertaln- 
tles  can  result  in  design  changes  and  corresponding  tooling  changes. 
Unresolved  uncertainties  subsequent  to  the  OPEVAL  period  produce  a 
significantly  greater  Impact  on  a system  in  terms  of  both  time  and 
dollars,  as  Figure  3-2  Illustrates.  The  nature  of  this  impact  is 
defined  in  the  flow  diagram  of  Figure  3-3. 

During  the  early  stages  of  system  installation  and  utilization 
in  the  operational  forces,  an  unresolved  problem  can  result  in 
costly  retooling  and  retrofit.  Furthermore,  the  Fleet’s  defense 
readiness  may  be  significantly  impaired  during  the  retrofit  period. 
The  alternative  to  modifying  the  system  is  a choice  to  live  with 
the  system  effectiveness  degradation  which  results  from  the  problem. 

Two  courses  of  action  may  be  presented  as  a result  of  a 
decision  to  Improve  a system’s  effectiveness  to  an  acceptable  level. 
One  action  is  to  make  Improvements  through  equipment  modifications, 
which  involves  a fixed  or  one-time  cost.  The  other  action  involves 
improving  the  effectiveness  through  logistic  or  personnel  changes, 
which  generates  recurring  costs.  An  example  of  personnel  changes 
would  be  where  system  effectiveness  is  Improved  by  the  establishing 
of  a higher  personnel  level  for  maintenance. 


3.2  Cost  Effectiveness  Criteria  for  a 
System  fevaluatlon  Facility 

In  section  3.1>  the  extent  of  system  evaluation  was  related  to 
the  expected  cost  of  ownership  and  delay  in  system  delivery.  These 
two  relationships  essentially  define  the  effectiveness  of  a system 
evaluation  facility.  The  delivery  of  an  acceptable  system  can  be 
delayed  if  the  evaluation  period  is  either  too  long  or  too  short. 

In  the  former  case,  the  resulting  decrease  in  OPEVAL  time  and  in 
the  period  of  unacceptable  operation  following  Installation  will 
not  compensate  for  the  extended  evaluation  time.  In  the  latter 
case,  the  advantages  of  early  physical  delivery  of  the  item  will  be 
negated  by  the  extended  OPEVAL  and  unacceptable-operation  period. 

^Delivery  refers  here  to  general  acceptance  of  the  system  by  the 
Fleet,  rather  than  physical  delivery. 
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Expected  Cost  to  Make  a Major  Change  in  a 
Typical  Shipboard  Electronic  System  (Dollars) 


Time  delays  cannot  be  readily  expressed  in  terms  of  dollars 
unless  some  rather  restricting  assumptions  are  invoked.  Therefore, 
both  the  cost  of  system  ownership  (CO)  and  delay  in  system  delivery 
(DD)  will  be  used  as  measures  of  the  effectiveness  of  the  system 
evaluation  facility. 

The  expected  cost  of  ownership  of  a system  (which  may  or  may 
not  utilize  the  system  facility)  can  be  exp""essed  as 


A(CO)  = (CO)g^  ^ 


■^min 


(CO); 


Et-  ^ E 


'min 


(3-1) 


where  A (CO) 


^ E. 


'min 


Expected  influence  on  total  system  cost  of 
ownership  of  a given  amount  of  system- level 
evaluation 

Total  cost  of  ownership  without  a system  eval- 
uation, but  for  a delivered  product  whose 
Inherent  effectiveness  equals  or  exceeds  the 
minimum  requirement 


(CO) 


■ ^mln 


Total  cost  of  system  ownership  with  an  adequate 
system  evaluation* 


*The  assumption  was  made  here  that  an  adequate  system  evaluation 
is  one  which  will  result  in  a normal  OPEVAL  period  and  insignifi- 
cant engineering  changes  (ECs)  to  the  Installed  systems. 
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Both  (C0)„  i „ and  (C0)„  ^ _ contain  a common-base 
“ ^In  “ ^In 

cost  of  ownership,  X,  which  for  this  analysis  is  only  of  academic 
interest  since  cost  differences  serve  as  the  comparative  measure. 
In  terms  of  this  common  cost,  the  two  expressions  can  be  further 
defined  as: 


A(CO)j,^ 


s E 


'min 


^ ^EC  '^^OPEVAL 


(3-2) 


where 

AC 

*^syst . 
Thus, 


^ ^syst.  eval. 

(3-3) 

^EC 

= Accumulated  cost  of  engineering  changes  which 
occur  both  as  a result  of  OPEVAL  and  problems 
uncovered  during  operational  use 

OPEVAL 

= Cost  of  conducting  OPEVAL  beyond 
period 

the 

planned 

eval. 

= Cost  of  system  evaluation 

equation  3-1  reduces  to: 


A(CO) 

The  delay  in  delivery  of  an  acceptable  system  can  be  similarly 
treated  (this  time  omitting  the  subscript  Ej  s Ejj^j_j^,  which  still 
applies): 

A(DD)  ==  DD  - DD'  (3-3) 

where  A(DD)  = Expected  influence  on  system  delivery  time  of  a 

given  amount  of  system- level  evaluation 

DD  = Delay  in  delivery  of  an  acceptable  system  which 
has  not  undergone  a system  evaluation 

DD*  = Delay  in  delivery  of  an  acceptable  system  with 
an  adequate  system  evaluation 


The  factors  DD  and  DD*  can  each  be  expanded  in  terms  of  a 
common  time  lapse,  T,  plus  delays  unique  to  each: 


DD  - T + ATp^  + ATqpj^^j^ 

(3-4) 

DD  - T + ^ eval . 

(3-5) 
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Accumulated  time  of  Interruption  of  fleet- 
readiness of  a ship  while  engineering  changes 
are  being  identified  and  installed;  divided  by 
the  number  of  ships  involved 

Delay  In  delivery  caused  by  stretch-out  of  OPEVAL 

Net  amount  of  development  time  attributed  to 
adequate  system  evaluation,  including  changes 
and  redirection  resulting  therefrom 

In  examining  the  cost  effectiveness  of  a system  evaluation 
facility,  the  pro-rated  direct  cost  (see  section  6)  of  the  evalua- 
tion for  a given  system  must  be  compared  with  the  potential  Impact 
upon  total  system  cost  of  ownership,  Aco,  and  delay  in  delivery, 
add,  of  an  acceptable  product.  The  latter  measures  of  effective- 
ness, as  previously  defined  in  equations  3-1  and  3-3>  are  identified 
as  the  risks  in  the  decision  matrices  of  sections  4 and  5- 

3.3  Development  of  Facility  Cost  Comparisons 

The  cost  structure  which  was  formulated  for  comparison  of 
the  alternative  facilities  is  shown  in  Figure  3-4.  Acquisition 
cost  is  that  associated  with  making  a facility  available  and  ready 
for  use,  and  is  a one-time  fixed  amount  occurring  at  the  beginning 
of  the  term  of  ownership.  Operation  cost  includes  the  funds 
necessary  to  operate,  maintain,  and  administer  the  facility,  and 
is  calculated  on  a per-annum  basis. 


where  = 


^'^OPEVAL 


T 

syst . eval. 


FIGURE  3-4 
COST  STRUCTURE 
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3. 3-1  Basis  for  Estimating  Facility  Costs 

The  procedures  described  In  a recent  ARINC  Research  report* 
served  as  a basis  for  estimating  facility  costs.  These  procedures 
were  originally  derived  to  evaluate  the  cost  effectiveness  of 
shipboard  electronic  equipment;  however,  they  have  equal 
applicability  to  a system  test  facility. 

In  comparing  the  alternative  facilities,  only  those  costs 
occurring  in  the  future  are  considered  of  practical  value;  all 
past  expenditures  are  considered  as  sunk  costs.t  In  this  study, 
costs  are  examined  for  the  ten-year  period  from  June  1966  to 
June  1976.  The  decision  to  examine  costs  over  a ten-year  period 
was  somewhat  arbitrary;  however,  a twenty-year  interval  was  also 
examined  to  ensure  that  no  significant  alternative- selection 
differences  occurred.  The  shorter  interval  was  favored  because, 
from  a management  plajinlng  viewpoint,  the  visibility  beyond  a 
ten-year  period  taxes  the  ability  to  quantitatively  assess  the 
requirements  for  a system  facility. 

3.3.2  Modified  Cost  of 
Ownership 


A special  effort  was  made  to  minimize  the  possibility  of 
biasing  any  of  the  alternatives.  Therefore  a cost-analysis  method 
was  developed  which  is  equally  meaningful  whether  used  on  a ship, 
hull,  building,  or  battery.  Likewise,  even  though  the  incremental 
costs  were  not  precisely  determinable,  estimates  were  made  based 
on  the  same  logic  for  each  alternative.  With  this  approach  the 
modified  cost  of  ownership  of  the  various  alternatives  could  be 
realistically  compared,  even  though  the  absolute  costs  had  not 
been  estimated. 


3.3.3  Effect  of  Requirements  on 
facility  Costs' 

Since  a decision  was  made  to  select  the  best  of  several 
alternative  facilities  for  testing  shipboard  communication  and 
other  electronic  systems,  some  minimum  acceptable  characteristics 
were  selected.  Table  2-1  lists  both  minimum  acceptable  and  desir- 
able characteristics  for  the  most  important  requirements.  Desir- 
able characteristics  over  and  above  these  minimums  were  considered 


*Cost-Effectiveness  Evaluation  Procedures  for  Shipboard  Electronic 
Equipment  and  Systems.  ARINC  Research  Publicatjon  SOQ-Ol -p-RfSA, 
February  1966,  two  volumes . 

This  principle  is  treated  in  detail  in  Principles  of  Engineering 
Economy,  E.  L.  Grant  and  W.  G.  Ireson,  Ronald  Press  Co.,  N.  Y., 
Fourth  Edition,  1964. 
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as  irreducible  factors.  In  this  way  the  basic  calculations  are 
partially  normalized,  but  the  Irreducible  factors  (e.g.  expansion 
flexibility)  are  still  considered  in  the  final  decision. 

3.3.^  Irreducible  Factors 

Although  most  of  the  differences  between  alternative  facilities 
can  be  expressed  in  terms  of  cost,  some  differences  relevant  to  the 
decision  cannot  be  so  reduced.  The  more  significant  irreducible 
factors  for  the  alternative  facilities  are  listed  in  this  section.* 

3. 3.4.1  Bunker  Hill  at  Pier  F 

(1)  Facility  available  for  immedj'>te  use  with  a short  lead 
time  for  equipment  installation. 

(2)  Mobility:  can  be  moved  to  any  location  in  a waterway. 

(3)  Can  be  used  as  a portable,  emergency  communication  station 

(4)  Can  be  used  as  a ship's  security  laboratory. 

(5)  Can  be  used  as  a shipboard-noise  analysis  laboratory. 

(6)  May  be  a hazard  to  the  operation  of  aircraft  from  NAS, 

North  Island,  and  seaplane  lanes. 

(7)  Industrial  safety  hazard  greater  aboard  ship. 

(8)  Working  conditions  may  be  only  marginally  acceptable. 

(9)  Psychological  advantage  of  shipboard  environment  for 
personnel  accustomed  to  ships. 

(10)  Can  be  used  as  an  operation/maintenance  training  facility, 
both  generally  and  as  proposed  by  the  0-in-C  Naval  Radio 
School,  NTC,  San  Diego. 

(11)  Facility  is  reportable  under  the  Navy  Cost  Reductlpn 
Program  to  the  Secretary  of  Defense  under  the  category 
of  utilization  of  surplus  materials. 


*Such  factors  a "lead  time"  to  establish  and  Implement  a Military 
Construction  Project  (MILCON)  may  be  borderline  and  partially 
considered  as  reducible  and  partly  as  irreducible. 
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3.3*^-2  Bunker  Hill  Adjacent  to 
East  Shore  of  Pt.  Loma 


(1)  Facility  available  for  iiranedlate  use  with  a short  lead 
time  for  equipment  installation. 

(2-11)  Same  as  foregoing  items  1-11  (section  3. 3. 4.1),  except 
item  6 not  applicable. 

(12)  Some  delay  and  expense  Involved  in  preparation  of  site. 
However,  delay  would  be  slight  during  transit  to  Pt . Loma 
since  operation  could  continue  at  Pier  F until  new  site 
is  ready. 

(13)  Berthed  close  to  the  cliffs  south  of  Ballast  Point,  it 
will  be  less  conspicuous  than  at  Pier  F. 

(14)  Productive  use  will  be  made  of  valuable,  unused  harbor 
property. 

(15)  A prior  claim  will  be  established  on  valuable,  unused 
harbor  property. 

(16)  Better  physical  control  of  the  facility  is  afforded  by 
location  as  a part  of  the  Pt . Loma  reservation.  Better 
fire  protection  and  emergency  maintenance  support,  etc. 

3. 3*4. 3 West  Side  of  Pt.  Loma, 

Water-Level  Building 

(1)  Proximity  to  water  partially  approximates  the  radiatioii 
ground  plane  of  a ship . 

(2)  Access  road  has  steep  gradient. 

(3)  Industrial  safety  features  can  be  built  in. 

(4)  Long  lead  time  to  complete  building  (at  least  three 
years) . 

(5)  Cannot  meet  Fleet  School  or  Communications  Security 
support  requirements. 

(6)  New  building  has  a high  aesthetic  value. 

3. 3*4. 4 Compound  Building 

(1)  Flans  are  to  use  part  of  a building  presently  under  a 
MILCON  project. 

(2)  Long  lead  time  to  complete  building  (at  least  three 
years) . 
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(3)  Designated  portion  of  building  (35,750  ft^)  can  be 
designed  for  specific  use. 

(4)  Forms  Integral  part  of  present  laboratory. 

(5)  New  building  has  a high  aesthetic  value. 

3.3*4.5  Battery  Ashburn 

(1)  7,000  ft^  could  be  made  available  for  immediate  use. 

(2)  Remaining  28,750  ft^  will  require  long  lead  time  to 
build . 

(3)  Secure  spaces  presently  available. 

(4)  Present  occupants  must  be  m.oved  elsewhere. 

(5)  Cannot  meet  Fleet  School  or  Communications  Security 
support  requirements. 

(6)  Integration  of  existing  space  with  new  construction 
poses  practical  problems  in  facility  design  and 
operation. 

(7)  New  construction  would  be  adjacent  to  unrestricted 
public  thoroughfare. 

3.3*4. 6 Leased  Building 

(1)  Short-term  lease  allows  flexibility  to  discontinue  or 
move  facility. 

(2)  Low  acquisition  costs. 

(3)  Location  may  be  inconvenient. 

(4)  Security  specification  may  be  difficult  to  meet. 

(5)  Least  accessible  for  material  and  personnel  movement. 

(6)  May  pose  difficult  and  expensive  problems  of  data 
linkage  to  existing  NEL  facilities. 

(7)  Cannot  meet  Fleet  School  or  Communications  Security 
support  requirements. 

(8)  Proximity  to  public  thoroughfares. 
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4.  RELATIVE  EFFECTIVENESS  OF  FACILITY  ALTERNATIVES 

The  relative  effectiveness  of  the  six  alternative  facilities 
was  the  subject  of  considerable  study.  The  basic  factors  In  the 
study  were  the  evaluation  objectives  for  which  the  facility  Is 
being  planned,  and  the  projected  risks  associated  with  each  objec- 
tive. As  explained  in  section  3»  each  risk  has  an  Influence  on 
the  ownership  of  the  systems  which  are  likely  to  be  evaluated  at 
the  facility. 

Table  4-1  gives  a comprehensive  analysis  of  the  evaluation 
objectives  and  risks.  Including  an  indication  of  the  adequacy  of 
each  alternative  facility  with  respect  to  the  system  evaluation 
objectives.  (The  table  follows  page  4-6.) 

4.1  General  Assumptions 

In  the  decision  model  described  in  section  4.2,  certain 
general  assumptions  have  been  made;  these  assumptions  are  summarized 
below  to  assist  the  reader  in  interpreting  the  information  in 
Table  4-1. 

The  numerical  information  in  the  table  is  not  traceable  to 
specific  references  or  specific  contact  with  technical  personnel. 
Instead,  they  represent'  ARINC  Research  Corporation's  best  Judg- 
ment, based  on  all  available  information  (gained  primarily  from 
interviews) . 

During  the  period  of  interviewing  and  data  collection  which 
preceded  the  analysis,  provision  was  made  for  treating  the  effec- 
tiveness of  each  of  the  six  alternate  facilities  individually. 
However,  when  the  information  was  assembled  and  analyzed,  it  was 
found  that  from  a technical  standpoint  the  six  alternatives  could 
be  conveniently  grouped  into  two  categories:  "hull"and  "building". 
With  the  assumed  minimum  requirements*  set  forth  in  section  2.1, 
the  differences  in  technical  suitability  between  the  Bunker  Hill 
(hull)  at  Pier  F (LA)  or  south  of  Ballast  Point  (IB)  were  found 
to  be  insignificant.  Similarly,  the  differences  in  technical 
suitability  between  all  the  remaining  (building)  candidates  were 
found  insignificant.  Table  4-1  therefore  categorizes  the  alterna- 
tive facilities  as  either  hull  or  building  types.  Comparison 
within  these  two  categories  must  be  in  terms  of  either  direct 
costs  or  irreducible  factors. 

The  first  impression  during  preliminary  evaluation  is  that 
the  opportunity  to  utilize  a carrier-type  hull  from  the  reserve 
fleet  would  offer  am  overall  environment  of  unquestionable  realism 
from  a system  evaluation  standpoint.  However,  in  attempting  to 


*The  cost  of  providing  these  same  minimum  requirements  in  each 
case  is  included  in  the  cost  comparisons  of  section  6. 
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isolate  and  study  these  benefits  in  greater  detail,  the 
advantages  of  the  ship  become  less  pronounced.  Some  of  the 
leavening  considerations  are  as  follows: 


(1)  The  carrier  hull  and  compartment  configuration  is  not 

representative  in  all  respects  of  the  configurations  ' 

for  which  many  of  the  new  shipboard  systems  are  being 

planned.  In  addition,  the  carrier  hull  superstructure 

cannot  be  patterned  to  that  of  projected  ship  super-  1 

structures  without  incurring  unreasonable  costs.  How- 

ever,  the  Bunker  Hill' s compartment at ion  approximates 

most  hulls  except  for  command  centers.  These  centers 

could  be  duplicated  on  the  hangar  deck  with  a closer 

approximation  of  reality  than  can  be  achieved  in  a 

shore  building. 

(2)  The  electronic  suit  aboard  the  Bunker  Hill  hull  is  neither 

complete  enough  nor  sufficiently  updated  to  be  repre- 
sentative of  the  "other  system"  electromagnetic  environ-  i 

ment  projected  for  systems  currently  in  development. 

(3)  The  use  of  the  hull  as  a means  of  identifying  and 
studying  the  "rusty  bolt"  problem*  should  be  classed 
as  experimentation  rather  than  system- level  evaluation. 
Nevertheless,  the  hull  does  provide  a realistic  mech- 
anism in  establishing  a noise  floor  in  the  presence 
of  a system  installation.  Therefore,  there  is  good 
reason  to  evaluate  noise  components  and  validate  pre- 
diction models  in  actual  system- level  operation. 

(4)  Treatment  of  the  personnel  envlrormient  necessary  for 
operation  and  maintenance  evaluation  can  be  made  sur- 
prisingly realistic  in  any  building.  A good  example 
is  the  FAAWTC  operation  adjacent  to  NEL. 

In  spite  of  the  aforementioned  practical  problems,  certain 
advantages  of  conducting  the  system  evaluation  aboard  an  actual 
hull,  such  as  the  Bunker  Hill,  must  be  recognized.  These  can  be 
itemized  as  follows: 

(1)  A significant  value  can  be  actached  to  the  capability 
in  the  Bunker  Hill  for  installing  portions  of  a system 
in  a compartment  and  using  cable  runs  (signal  and  power) 
somewhat  representative  of  a realistic  installation. 

(2)  The  evaluation  of  security  problems  (particularly  the 
red-black  problem)  aboard  the  compartmented  Bunker  Hill 
would  be  more  valid  than  a comparable  study  in  a shore- 
based  building. 


*"Rusty  bolt"  problem:  the  re-radiation  of  harmonics  by  the 
nonlinear  steel  joints  under  parasitic  excitation. 
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(3)  With  respect  to  the  human  aspects  of  a given  system, 
the  Bunker  Hill  offers  a much  more  realistic  "duty" 
environment  than  does  a building — particularly  as  the 
training  phase  of  evaluation  is  reached  for  each 
system. 

(^)  Naturally  present  are  problems  associated  with  piping, 
cabling,  ducting,  bulkhead  structures,  and  other  ship- 
board accessories;  in  practice  these  physical  items  would 
tend  to  be  sacrificed  in  any  "budget  squeeze"  in  a shore 
building. 

It  has  been  assumed  in  this  section  that  to  meet  the  minimum 
RF  requirements  for  communications  systems,  both  Bunker  Hill  alterna- 
tives (lA  and  IB)  would  be  augmented  with  an  antenna  installation 
on  the  west  shore  of  Pt.  Loma  covering  VHF  and  higher  frequencies. 
This  installation  would  be  functionally  connected  to  the  ship  over 
a multi-channel  microwave  link.  Thus  the  communications  system 
aboard  the  Bunker  Hill  could  be  in  complete  contact  with  the  First 
Fleet  operating  off  the  West  Coast  through  either  the  antenna 
augmentation  or  through  the  live  circuit  of  FAAWTC  via  microwave. 

Other  assumptions  of  this  study  are  (1)  Bunker  Hill  will  not 
be  returned  to  the  Reserve  Fleet,  and  (2)  disposition  costs  will 
be  offset  by  surplus- item  and  salvage  returns.  However,  it  should 
be  recognized  that  disposition  costs  will  not  be  affected  by  these 
returns  on  an  appropriation  basis,  in  that  the  returns  are  not 
credited  to  the  account  burdened  by  disposition  costs.  Actually 
there  is  reasonable  expectation  that  the  sale  and  salvage  returns 
would  exceed  disposition  costs.  For  this  reason  the  second 
assumption  at  the  beginning  of  this  paragraph  has  been  applied  in 
the  analysis. 

4.2  Decision  Model 

Relative- effectiveness  considerations  for  the  six  alternative 
facilities  have  been  reduced  to  the  decision  matrix  of  Table  4-1. 
Column  headings  of  the  matrix  are  described  in  the  following  para- 
graphs. In  arriving  at  the  numerical  entries  in  this  matrix, 
consideration  has  been  limited  primarily  to  that  family  of  ship- 
board electronic  systems  which  characterize  the  new  era  in  Navy 
operations.  Representat e of  this  category  are  the  integrated 
communications  system,  the  tactical  data  systems,  the  intelligence 
systems,  and  the  strategic  data  systems.  Among  the  common  proper- 
ties of  these  systems  are  computer-centered  operation,  with  high- 
volume  digital  information  flow;  and  the  fact  that  each  system 
will  eventually  Interface  with  all  of  the  others. 

Referring  to  Table  4-1,  the  first  column  contains  the  three 
basic  components  of  system  effectiveness — Performance,  Availability, 
and  Utilization.*  The  second  column  presents  a numerical  breakdown 


*The  lumbers  before  and  after  these  effectiveness  components  will 
be  explained  later. 
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of  each  of  these  three  components.  The  third  column  contains  a 
statement  of  an  evaluation  objective  appropriate  to  the  class  of 
systems  for  which  system  evaluation  is  planned.  The  fourth 
column,  "System  State",  contains  a weighting  factor  representing  the 
probability  (relative  to  other  effectiveness  subcategories)  that 
a new  system  concept  will  have  a significant  problem  area  or 
uncertainty  attendant  to  its  development.  The  numbers  for  this 
column  were  derived  by  the  process  of  starting  with  the  three 
major  components  of  effectiveness  (Performance,  Availability,  and 
Utilization)  and  apportioning  weighted  numerical  values  (adding 
to  unity)  among  them  according  to  best  engineering  assessment  of 
pertinent  information.  The  apportionment  given  in  the  table 
reflects  the  judgment  that  system-design  problems  can  be  expected 
to  occur  with  the  following  relative  probabilities: 

Performance  — 0.5 

Availability  — 0.2 

Utilization  — 0.3 

For  example,  performance  problems  are  0.5/0. 2 = 2.5  times  as 
likely  to  occur  as  are  availability  problems. 

Some  of  the  experience  upon  which  these  assignments  are  based 
is  embodied  in  the  case  histories  constructed  by  ARINC  Research 
for  NEL  under  the  present  contract.*  Other  experience  accrued 
from  interviews  with  representatives  of  OPDEVFOR,  EMEC,  and  OEG. 

The  value  assigned  to  each  of  the  three  major  components  was 
similarly  apportioned  among  their  subcomponents.  This  apportion- 
ment process  was  chosen  to  avoid  serious  biasing  of  the  remainder 
of  the  matrix. 

The  fifth  column,  "Probability  of  Uncertainty  Remaining", 
contains  estimates  conditional  on  the  system  state  number.  This 
number  represents  the  probability  that'  the  problem  of  uncertainty 
(given  that  one  exists)  would  not,  through  the  due  course  of  sub- 
system development,  be  adequately  Identified  and  resolved. 

The  sixth  column,  "Risk  Category",  contains  an  index  repre- 
senting the  magnitude  of  consequences  resulting  from  failure  to 
properly  identify  and  resolve  a problem  area  or  uncertainty 
associated  with  each  evaluation  objective,  considered  one  at  a 
time.  Five  levels  of  risk  are  provided,  corresponding  to  powers 
of  the  base  five.  For  example,  the  number  1 in  this  column  refers 
to  a risk  of  five  to  the  first  power.  Similarly,  the  number  5 
refers  to  a risk  of  five  to  the  fifth  power.  This  stratification 
of  risk  is  intended  to  cover  the  range  of  consequence  from  the 
installation  of  simple  modification  kits  in  the  field  to  the 
removal  of  a ship  from  the  Fleet  for  overhaul.  In  terms  of 


*ARINC  Research  Corp.,  Role  of  Test  and  Evaluation  in  System 
Development,  Publication  W5-404-TN001-1,  November  I965. 
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dollars,  this  risk  stratification  might  be  expressed  according 
to  the  following  equivalents  table: 


Risk 

Category 

Relat ive 
Risk 

Dollar  Equivalence 

Cost  of  Design  and 
Installation  of: 

1 

$ 0.1 

X 

10® 

Simplest  modification  kit 

2 

52 

0.5 

X 

10® 

3 

53 

2.5 

X 

10® 

4 

12.5 

X 

10® 

5 

55 

62.5 

X 

10® 

Major  modification 

requiring  overhaul 
availability 


Obviously,  dollars  are  not  the  only  classification  of  risk. 
Two  other  classifications  are  important:  the  time  delay  factor 
(see  section  3)  a-^d  the  detrimental  effect  on  Fleet  readiness 
which  accompanies  any  modification  program.  The  latter  factor 
is  also  significant  as  an  element  of  surprise  in  experiencing 
degraded  performance  when  the  system  is  first  called  upon  to  per- 
form its  mission.  The  latter  two  classifications  have  been  con- 
sidered in  each  table  entry  and  have  been  subjectively  translated 
into  terms  of  dollars.  For  example,  a problem  for  which  the 
solution  is  low  in  cost  but  significant  in  terms  of  downtime  of 
the  ship  and  degradation  in  Fleet  readiness  will  be  assigned  a 
risk  level  considerably  higher  than  that  based  on  cost  alone. 

Column  seven  is  subdivided  into  two  columns  representing 
"hull"  and  "building"  type  facilities.  The  entries  in  these 
columns  represent  the  probability  of  the  indicated  system  facility 
alternate  providing  adequate  identification  and  resolution  of  the 
corresponding  system  uncertainty,  assuming  each  alternative 
satisfies  the  minimum  requirements  of  section  2.2. 

The  eighth  column,  "Modified  Risk  Index",  contains  entries 
obtained  from  other  numerical  entries  in  the  table  according  to 
the  following  formula: 

Modified  Risk  = System  State  x Probability  of  Problem  Remaining 

X (1  - Adequacy  of  Facility  Alternate) 

X 5 exp(Risk  Category). 

The  reader  is  cautioned  against  attaching  any  significance 
to  the  absolute  value  of  any  modified  risk  entry  or  to  the  sum- 
mations at  the  bottom  of  Table  4-1.  These  numbers  have  relat ive 
significance  only. 

The  right-hand  column,  "Qualifying  Remarks",  indicates  the 
reasoning  londerlying  the  choice  of  numbers  for  each  entry  in  the 
table. 
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4.3  Conclusion 


The  modified  risk  summation  at  the  bottom  of  Table  4-1  Is 
presented  as  an  Index  of  the  relative  effectiveness  of  the  six 
alternate  system  facilities.  Simply  speaking,  these  numbers 
represent  a flgure-of-merlt  rating  for  each  facility  according 
to  the  facility's  ability  to  Impact  on  the  life  cycle  of  a typi- 
cal complex  shipboard  electronic  system. 
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5. 


OVERALL  NEED  FOR  EVALUATION 


This  section  contains  an  elementary  exercise  in  decision 
theory  incorporating  the  experience  factors  used  in  the  relative 
effectiveness  matrix  (Table  4-1).  However,  in  this  section  the 
problem  is  reformulated  into  a framework  (Table  5-1)  which 
addresses  the  question  of  whether  or  not  system- level  evaluation 
is  justified.  The  analysis  is  based  primarily  on  the  categorization 
of  a system  state  into  one  of  three  groups,  weighted  from  experi- 
ence. The  three  groups  define  the  probable  acceptability  of  a new 
system  configuration  prior  to  OPEVAL  and  appear  as  horizontal  rows 
in  Table  5-1. 

Two  basic  actions  (test  or  no- test)  refer  to  whether  or  not 
a system- level  evaluation  is  performed  with  the  system.  The 
"no  test"  entries  correspond  to  the  consequences  during  OPEVAL 
and  subsequent  to  installation.  The  "test"  entries  correspond 
to  each  of  three  possible  test  results  (accept,  modify  and  retest, 
and  reject) . Again  the  consequences  of  each  outcome  during  sub- 
sequent OPEVAL  and  installed  operation  are  given.  In  each  case, 
the  consequences  are  stated  in  terms  of  dollars  and  delay  time. 

An  expectation  (or  mean  value)  for  each  consequence  is  obtained 
by  multiplying  the  dollar  or  time  figure  by  the  system  state 
probability,  and--for  "test"  only — by  the  probability  of  adequate 
system  evaluation. 

The  summation  of  expected  dollar  and  delay-time  risks  at  the 
bottom  of  the  figure  then  allows  a comparison  of  consequences,  on 
a probable  risk  basis,  corresponding  to  the  selected  action. 

The  numbers  in  Table  5-1  representing  cost  and  delay  time 
are  calculated  from  the  assumed  conditions  of  Table  5-2.  A sys- 
tem development  was  structured  in  this  table  which  could  be  typical 
of  the  sophisticated  Navy  systems  now  under  procurement  or  planned 
for  the  near  future. 

The  results  of  the  example,  as  indicated  at  the  bottom  of 
Table  5-1,  would  indicate  that  severe  consequences  in  both  dollars 
and  delivery  time  can  be  expected  when  any  significant  compromise 
is  made  in  the  application  of  a system  evaluation.  This  can  be 
seen  by  comparing  the  expected  risk  cost  of  not  performing  a system 
evaluation  ($31.69  million)  with  the  expected  risk  cost  of  an 
adequate  evaluation  ($8.58  million).  This  conclusion  would  not 
necessarily  hold  for  systems  of  low  complexity  or  small  production 
quant ity . 
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**OPTEVFOR  is  constrained  to  not  reject 
a major  system. 


TABLE  5-2 

COST  AND  DELAY  COMPONENTS  ASSUMED  FOR 
(See  Table  5-1) 

EXAMPLE 

EVALUATION  FACILITY  (COST) 

Physical  structure 

= $400K  (3  systems) 

= 

$167,000 

Manpower  at  15-man 
level 

= $375K  (3  systems) 

= 

125,000 

Instrumentation 

= 

200,000 

Subsystem  personnel 
at  10-man  level 
(per  system) 

= $250K  (per  system) 

Total 

250.000 

775.000 

1 

OPEVAL 


Assume  6-month  planned  OPEVAL 

2 years  for  marginal  system  accepted  during  system 
evaluation 

12  months  for  unacceptable  system  accepted  during 
system  evaluation 

Assume  OPEVAL  requires  technical  crew  of  10  men  at  $20K  per 
man  per  year 

6 months  stretch-out  = $100,000 
l8  months  stretch-out  = $300,000 

(Not  including  cost  of  ship  out  of  operational  state) 

SYSTEM 

Assume 

— 20  copies  to  be  delivered  at  approximately  $5M  per  copy 
— Normal  R&D  phase  costs  $30M 

— Redevelopment  costs  are  25  percent  of  original  R&D 
($7.5M) 

— Cost  of  engineering  changes  (ECs)  after  installation 
is  25  percent  of  original  cost,  or  $1.25M  per  system 

— Cost  of  engineering  changes  for  marginal  system  prior 
to  production  adds  5 percent  to  original  cost  (for 
redesign,  retooling,  and  minor  parts  reordering),  or 
$0.25M  per  system 

— Cost  of  engineering  changes  for  unacceptable  system 
after  installation  ($50M) 


COST  COMPARISON 


As  stated  in  section  3j  cost-of-ownership  techniques  were 
used  in  organizing  and  presenting  the  data;  however,  since  only 
those  costs  significant  to  a decision  were  considered,  the  results 
are  labeled  "Modified  Cost  of  Ownership".  After  a decision  is 
made  and  an  alternative  selected,  it  may  be  wise  to  continue  and 
develop  a Total  Cost  of  Ownership  using  the  previously  developed 
techniques.  This  would  require  a greater  depth  of  detail  in  the 
gathering  and  analysis  of  additional  data.  The  results  of  a 
cost  comparison  of  the  six  alternative  system-test  facilities 
are  given  in  Table  6-1. 


6 . 1 Cost  Details 

On  pages  6-3  through  6-7  the  costs  associated  with  the  acquisi- 
tion and  operation  of  each  alternative  facility  are  explained  in 
further  detail.  These  costs  are  representative  of  the  most  sig- 
nificant expenditures  that  may  influence  a decision.  Some  costs, 
such  as  the  equipment- installation  costs  common  to  all  alternatives, 
have  not  been  included.  Most  of  the  items  are  common  to  all  of 
the  facilities  (e.g.  utilities,  parking,  etc.).  However,  some 
are  unique  to  one  or  two  alternatives,  such  as  cathodic  protection, 
dehumidification,  and  conductive  roofs. 

Since  most  of  the  costs  are  estimates  they  have  been  rounded 
off  to  the  most  significant  digit.  All  known  sources  of  relevant 
information  were  surveyed,  and  the  data  were  carefully  examined 
for  validity.  No  attempt  was  m.ade  to  establish  a confidence  level 
for  each  item  since  there  was  insufficient  time  to  examine  all  the 
design  details  of  each  alternative.  It  is  suggested  that  in  cases 
where  the  reader  questions  an  input,  a new  estimated  cost  be 
inserted  and  examined  for  significance.  If  it  appears  significant, 
verification  in  more  detail  may  be  in  order. 


6.2  Summary  of  Facility  Costs 

For  each  of  the  candidate  facilities.  Table  6-1  summarizes 
the  cost  of  acquisition  and  operation  over  a ten- year  period 
(June  1966-June  1976).  As  explained  in  section  3.3-li  funds 
expended  prior  to  June  1966  (sunk  costs)  have  not  been  considered. 
The  structure  acquisition  (1100)  is  low  for  alternatives  lA  and  IB 
(Bunker  Hill)  because  the  basic  structure  exists  and  a compart- 
mentatlon  set-up  is  all  that  is  required  on  the  hangar  deck. 
Similarly,  since  3B  is  a leased  building  the  original  costs  are 
low.  The  cost  of  lease  pa3^ents  over  a ten- year  period  are  shown 
as  a double  entry  under  2200/3B.  Note  that  when  costs  alone  are 
considered  over  a ten-year  period,  the  order  of  preference  is  3A, 
2B,  lA,  2A,  IB,  and  3®.  However,  the  facility  effectiveness  and 
irreducible  factors  will  necessarily  influence  the  final  decision. 
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TABLE  6-1 


COST  OF  ALTERNATIVE  TEST  FACILITIES, 
JUNE  1966  THROUGH  JUNE  1976 


Acquisition  (1000) 

Operation  (2000) 

Facility 

(§2.2.2) 

Structure 

(1100) 

Site 

(1200) 

Admin- 

istration 

(2100) 

Mainte- 
nance and 
Support 
(2200) 

Total 

$ 

$ 

$ 

$ 

$ 

Bunker 

Hill, 

Pier  F 
(lA) 

325,000 

34,500 

1,000,000 

2,084,000 

3,443,500 

Bunker 

Hill, 

Pt . Loma 
(IB) 

325,000 

1,213,800 

200,000 

1,884,000 

3,622,800 

Water- 

Level 

Bldg. 

(2A) 

1,191,500 

688,910 

200,000 

1,422,750 

3,503,160 

Compound 

Bldg. 

(2B) 

1,191,500 

243,750 

200,000 

1,422,750 

3,058,000 

Battery 

Ashburn 

(3A) 

896,500 

232,950 

200,000 

1,422,750 

2,752,200 

Leased 

Bldg. 

(3B) 

350,000 

5,000 

1,000,000 

1,422,750 

+858,000* 

3,635,750 

*Ten-year  lease  cost . 


6.3  Expansion  Costs 

The  effect  on  cost  of  expanding  the  system  evaluation  facility 
to  double  size  after  five  years  is  illustrated  in  Figures  6-1 
and  6-2.  Compared  in  this  regard  are  the  Bunker  Hill  at  Pier  F and 
the  Battery  Ashburn.  (These  alternatives  were  selected  because  the 
former  represents  the  lowest  cost  of  expansion  and  the  latter  repre- 
sents the  lowest-cost  building.) 

Figure  6-1  shows  comparative  costs  over  a ten- year  period 
assuming  no  expansion  in  facility  size;  Figure  6-2  reflects  an  expan- 
sion to  double  size  after  five  years.  It  can  be  seen  that  the  cross- 
over point  of  cumulative  cost  shifts  from  slightly  more  than  5 years 
to  almost  9 years  if  expansion  occurs. 
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COST  DETAIL 
ALTERNATIVE  NO.  lA 
BUNKER  HILL,  PIER  F 


Structure  (1100) 


1. 

Designing  and  planning 

$ 75,000 

2. 

Construct  hangar  deck  compartments 

200,000 

3. 

Activate  compartments 

25,000 

4. 

Secure  area  5000  ft^  @ $5/ft^ 

25,000 

$325,000 

Site  (1200) 


1. 

Prepare  Pier  F for  light  vehicular 

traffic 

$ 8,000 

2. 

Connection  for  water  main 

4,000 

3. 

Parking  lot,  100  cars 

18,500 

4. 

Connection  for  sewerage  line 

4,000 

$ 34,500 

Administration  (2100) 

10 

civilians  @ lOK 

$100,000 

Maintenance  and  Support  f2200) 

1. 

Janitorial  service  (4  @ 6k) 

$ 24,000 

2. 

Upkeep  and  painting 

20,000 

3. 

Repairs 

20,000 

4. 

Cathodic  protection 

10,000 

5. 

Maintenance  supplies 

10,000 

6 . 

Utilities 

Electricity,  water,  sewerage. 

gas 

40,000 

Telephone 

14, 400 

7. 

Operation  of  shuttle  boat 

20,000 

8. 

Guards  (5  @ 8k) 

40,000 

9. 

Dehumidification  maintenance 

10,000 

$208,400 
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, COST  DETAIL 
ALTERNATIVE  NO.  IB 
BUNKER  HILL,  E.  SIDE  FT.  LOMA 


structure  ^1100) 

Same  as  Alternative  Nc  . lA 

$ 325,000 

Site  (1200) 

1.  Prepare  berth 

$1,000,000 

2.  Pedestrian  walkway 

50,000 

3.  Parking  lot,  100  cars 

18,500 

4.  Bring  in  Utilities 
a.  Electrical 

Cable 

42,400 

Overhead  reinforcement 

12,800 

Substation 

70,600 

b.  Water 

4,900 

c . Sewerage 

4,000 

d.  Communications 

10,600 

$1,213,800 

Administration  f2100^ 

2 civilians  @ lOK 

$ 20^000 

Maintenance  and  Support  C2200) 

Same  as  Alternative  lA 

$ 208,400 

Less  cost  of  shuttle  boat 

- 20,000 

$ 188,400 
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COST  DETAIL 
ALTERNATIVE  NO.  2A 
WEST  SIDE  WATER-LEVEL  BLDG. 


Structure  (1100) 


1. 

Standard  area,  30,750  @ $30/ft^ 

$ 

922,500 

2. 

Secure  area,  5000  ft^  @ $45/ft^ 

225,000 

3. 

Special  conductive  Surface  on  roof 

44,000 

$1,191,500 

Site  (1200) 

1. 

Grad ing 

$ 

240,000 

2. 

Drainage  structures 

11,700 

3. 

Paving 

32,700 

4. 

Utilities  (for  building) 

a.  Electrical,  underground 

50,880 

b.  Water 

5,880 

c.  Sewerage 

90,000 

d . Communications 

18,400 

5. 

Access  roads  / 

210,900 

6. 

Parking  area 

18,500 

7. 

Fenc ing 

4,950 

8. 

Landscaping 

5.000 

$ 

688,910 

Administration  (2100) 

2 

civilians  @ lOK 

$ 

20,000 

Maintenance  and  Supply  (2200) 

1. 

Janitorial  service — $0 . 50/f t^ /year 

$ 

17,875 

for  35,750  ft^ 

2. 

Upkeep 

10,000 

3. 

Repairs 

10,000 

4. 

Maintenance  supplies 

10,000 

5. 

Utilities 

a.  Water,  sewerage,  gas 

40,000 

b.  Telephone 

14,400 

6. 

Guards  (5  @ 8k) 

40.000 

$ 

142, 275 
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COST  DETAIL 
ALTERNATIVE  NO.  2B 
COMPOUND  BUILDING 


Structure  (1100) 


1. 

Building,  30,750  ft^  @ $30/ft® 

(standard  area) 

$ 

922,500 

5000  ft^  @ $45/ft®  (secure  area) 

225,000 

2. 

Special  conductive  treatment  of  roof 

44, 000 

$1,191,500 

Site  (1200^ 

1. 

Grading 

$ 

75,000 

2. 

Utilities  (for  building) 

a.  Electrical 

Underground  cable 

42,400 

Overhead  reinforcement 

12,800 

Substation 

70,600 

b.  Water 

4,900 

c . Sewerage 

4,000 

d.  Communications 

10,600 

3. 

Parking  area  (no  access  road  required) 

18,500 

4. 

Fenc ing 

$ 

4,950 

243,750 

Administration  (2100) 

Administrative  staff — 2 civilians  @ lOK 

$ 

20,000 

Maintenance  and  Support  (2200) 

1. 

Janitorial  service — $0.50/ft^  year 
for  35,750  ft® 

$ 

17,875 

2. 

Upkeep 

10,000 

3. 

Repairs 

10,000 

4. 

Maintenance  supplies 

10,000 

5. 

Utilities 

a.  Water,  sewerage,  gas 

40,000 

b . Telephone 

l4,4oo 

6 . 

Guards  (5  @ 8k) 

$ 

40,000 

142,275 

COST  DETAIL 
ALTERNATIVE  NO.  3A 
BATTERY  ASHBURN 


Structure  (1100^ 

1. 

Assume  7000  ft^  now  applicable 
and  available 

$ 

0 

2. 

Convert  2000  ft^  of  undeveloped  area 

50,000 

3. 

Construct  26,750  ft^  building  on  top 

802,500 

4. 

Special  conductive  treatment  of  roof 

$ 

44.000 

896,500 

Site  (1200) 

1. 

Grading 

$ 

90,000 

2. 

Utilities — expand  to  new  requirements 

110,000 

3. 

Expand  parking  area 

13,000 

4. 

Fencing 

4,950 

5. 

Landscaping 

5,000 

6. 

Access  road 

$ 

10,000 

232,950 

Administration  (2100) 

Administrative  staff — 2 civilians  @ lOK 

$ 

20,000 

Maintenance  and  Support  (2200) 

1. 

Janitorial  service — $0 .50/ft^/year 
for  35,750  ft^ 

$ 

17,875 

2. 

Upkeep 

10,000 

3. 

Repairs 

10,000 

4. 

Maintenance  supplies 

10,000 

5. 

Utilities 

a.  Water,  sewerage,  gas 

40,000 

b . Telephone 

l4,4oo 

6. 

Guards  (5  @ 8k) 

40.000 

$ 142,275 
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COST  DETAIL 
ALTERNATIVE  NO.  3B 
LEASED  BUILDING 


Structure  (1100) 


1.  Modify  5000  ft^  @ $15/ft®  for  security 

$ 

75,000 

2.  Designing  and  planning 

75,000 

3.  Make  compartments 

200,000 

First  Cost  (Fixed) 

$ 

350,000 

Lease  fee:  35 i 750  ft^  @ $2 . 40/f t^ /year 

$ 

85.800 

Site  (1200) 

Landscaping 

$ 

5,000 

Administration  (2100) 

Administrative  staff — 10  civilians  @ lOK 

$ 

100,000 

Maintenance  and  Support  (2200) 

1.  Janitorial  service — $0 .50/ft^/year 

$ 

17,875 

for  35,750  ft^ 

2.  Upkeep 

10,000 

3.  Repairs 

10,000 

4.  Maintenance  supplies 

10,000 

5.  Utilities 

a.  Water,  sewerage,  gas 

40,000 

b . Telephone 

l4,4oo 

6 . Guard  s ( 5 @ 8k) 

40,000 

$ 

142,275 
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$3,443,500 


Years  of  Operation 
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7. 


FINANCIAL  SUPPORT  OF  FACILITY 


In  an  attempt  to  maintain  an  objective  study  approach,  the 
cost  and  effectiveness  analysis  has  been  focused  to  a Navy-wide 
frame  of  reference.  Nevertheless  it  is  recognized  that  NEL 
management  decisions  must  take  account  of  the  practical  as  well  as 
the  altruistic  viewpoint — decisions  in  the  "best  interest  of  the 
Navy"  do  not  necessarily  assure  financial  support  for  the  facility. 
The  following  discussion  offers  a rationale  for  evaluating  this 
problem  area  and  suggests  an  approach  to  solving  the  financial 
problem. 

The  system  evaluation  facility  is,  by  nature,  a multiple- 
user  activity — in-house  as  well  as  non- NEL.  Although  initiated 
by  the  NSACS  group,  the  facility  will  clearly  have  utility  for 
other  laboratory  segments.  Two  specific  out-of-house  users  are 
known:  the  Radioman  Class  A/B  Schools  and  the  Naval  Security 
Group.  The  Personnel  Research  Laboratory,  other  Naval  Material 
Command  laboratories,  and  OPTEVFOR  are  other  potential  users  of 
the  facility. 

In  view  of  such  widespread  demand,  it  is  not  equitable  that 
NEL  bear  the  full  burden  of  financial  support  of  the  facility  and 
associated  administrative  costs.  It  is  suggested  that  the  facility 
could  be  operated  by  NEL  as  quasi-self-supporting.  Users,  whether 
in-house  or  non- NEL,  would  be  required  to  directly  fund  any  man- 
power and/or  materials  used  directly  in  their  utilization.  In 
addition,  users  would  be  required  to  fund  a share  of  the  general 
administrative  and  overhead  costs,  and  the  facility  maintenance 
and  utilities  cost,  prorated  on  the  basis  of  the  proportion  of 
the  facility's  capability  employed.  It  is  recognized  that  such 
an  approach  puts  a burden  on  NEL  to  assure  maximum  utilization  of 
the  facility  to  avoid  funding  the  overhead  burden  of  the  unused 
portion  of  the  facility.  However,  from  a management  point  of 
view,  this  could  provide  strong  motivation  for  maintaining  a high 
utilization  rate. 

This  approach  to  financial  support  of  the  facility  is  a 
modified  version  of  Navy  Industrial  Fund  (NIF)  operation.  The 
NEL  accounting  is  currently  on  a modified  NIF  basis.  As  a 
result,  this  approach  would  not  Introduce  undue  difficulty  in 
accounting,  other  than  to  require  the  setting  up  of  a separate 
account  for  the  facility.  Table  7-1  shows  comparative  ten- year 
costs  for  25  percent,  50  percent,  75  percent,  and  90  percent 
utilization  of  the  facility  alternates  under  this  suggested 
approach  to  financial  support. 

Successful  application  of  this  approach  would  be  contingent 
upon  establishing  the  percent  of  total  capability  represented  by 
the  various  segments  of  the  facility  as  a basis  for  determining 
the  charges  to  be  assigned  to  the  users.  However,  once  a 


schedule  is  established,  it  need  not  be  modified  unless  significant 
facilities  changes  are  made.  Study  might  indicate  the  feasibility 
of  a schedule  as  simple  as  one  based  on  square- foot age. 


TABLE  7-1 

COMPARATIVE  FACILITY  COSTS 
FOR  DIFFERENT  PERCENTAGES  OF  UTILIZATION 


Alter- 
nat ive 
(§2.2.2) 

Acquisition 

Cost 

NEL  Maintenance  and  Support  Costs  (10  years) 

25  Percent 
Utility 

50  Percent 
Utility 

75  Percent 
Utility 

90  Percent 
Utility 

$ 

$ 

$ 

$ 

$ 

lA 

359,500 

2,313,000 

1,542,000 

771,000 

308,400 

IB 

1,538,800 

1,563,000 

1,042,000 

521,000 

208,400 

2A 

1,880,400 

1,217,068 

811,375 

405,682 

162,275 

2B 

1,435,250 

1,217,068 

811,375 

405,682 

162, 275 

3A 

1,129,450 

1,217,068 

811,375 

405,682 

162,275 

3B 

355,000 

2,460,563 

1,640,375 

820,187 

328,075 
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